A Probabilistic Approach to ToF and Stereo Data Fusion

Current 3D video applications require the availability of high quality depth information. Depth information can be acquired real-time by stereo vision systems and ToF cameras. Both solutions present critical issues, that can be overcome by their combined use. In this paper, a heterogeneous acquisition system is considered, made of two high resolution standard cameras (stereo pair) and one ToF camera. The stereo system and the ToF camera must be properly calibrated together in order to operate jointly. Therefore this work introduces first a generalized multi-camera calibration technique which does not exploit only the luminance (color) information, but also the depth information extracted by the ToF camera. A probabilistic ad hoc fusion algorithm is then derived in order to obtain high quality depth information from the information of both the ToF camera and the stereo-pair. Experimental results show that the proposed calibration algorithm leads to a very accurate calibration suitable for the fusion algorithm, that, in turn, allows for precise extraction of the depth information.